Seeing Physics, or: Physics is for Prediction

We describe how knowledge of the physics of the scene itself is important to computer vision. High-level knowledge of scene physics can help programs see the world, and programs that see and understand this way are useful for planning plan scene interactions. We illustrate these points with two of our most recent knowledge-intensive vision systems. One uses knowledge of physics and function to understand noisy and ambiguous images of gear-train machines; i.e. to report what the machine does. The other uses physical knowledge to guide a robotic eye-hand system to pick up a mug of cooee by its handle.